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WHAT IS CLAIMED IS: 

1. A system for verifying test answers and providing 

collaborative scoring of the test answers between two 
or more test resolvers if the test answers contain a 
discrepancy, the system comprising: 

a) first receive means for electronically receiving 
an answer to a test question, the answer 
comprising an electronic representation of at 
least a portion of a test answer sheet; 

b) transmit means for electronically and 
independently transmitting the answer to first and 
second test resolvers; 

c) second receive means for electronically receiving 
first and- second test scores from the first and 
second test scorers; and 

d) verification means for electronically comparing 
the first and second test scores and for 
selectively recording a value related to the first 
and second test scores if the first and second 
test scores did not contain a discrepancy based on 
predefined criteria. 



2. TJie system of claim 1 further comprising display means 
for electronically displaying the test question. 
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3. The system of claim 1 wherein the verification means 
further comprises collaboration means for providing an 
electronic communication link between the first and 
second test resolvers and, if the first and second 
scores contain the discrepancy based on the predefined 
criteria, for signalling the first and second test 
resolvers to interactively determine and enter a joint 
score/ the verification means further comprising 

recording means for recording the joint score. 

- ■ 

4. The system of claim 3 wherein the collaboration means 
further comprises means for tracking average scores 
entered by the first and second resolvers and for 
preventing the first and second resolvers from scoring 
the same test item if the first and second resolvers 
each have average scores within predefined scoring 
criteria. 

5. The system of claim 3 wherein the verification means 
further comprises means for preventing the first and 
second test resolvers from receiving another answer to 
score until the first and second test resolvers have 
entered the joint score. 

6. The system of claim 3 wherein the verification means 
comprises third resolver means for transmitting the 
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test item to a third test resolver if the first and 
second test scorers do not enter the joint score within 
a predetermined time period, 

7. The system of claim 1 wherein the verification means 
further comprises third resolver means for transmitting 
the answer to a third test resolver f if the first and 
second scores contain the discrepancy based on the 
predefined criteria, and for recording a third test 
score received from the third test resolver. 

8. The system of claim 7 wherein the third resolver means 
further comprises means for providing the third test 
resolver with the first and second scores. 

9. The system of claim 7 wherein the third resolver means 
further comprises means for preventing the third test 
resolver from viewing the first and second scores 
before the third test resolver records the third test 
score. 
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10. The system of claim 1, wherein the verification means 
further comprises means for electronically linking the 
first and second test resolvers when the first and 
second test resolvers are located in different 
geographical locations . 
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11. The system of claim 1 wherein the verification means 
comprises means for recording at least one of a greater 
value of the first and second test scores , a lower 
value of the first and second test scores, and an 
average value of the first and second test scores. 

12. A system for evaluating the performance of test 
resolvers by routing quality-control questions for 
scoring to test resolvers, the quality-control 
questions comprising a test answer with a score known 
to quality-control personnel, the system comprising: 

a) transmit means for electronically transmitting to 
a test resolver an unidentified quality-control 
question comprising a test answer with a score 
known to quality-control personnel; 

b) receive means for electronically receiving from 
the test resolver a score for the quality-control 
question; and 

c) processing means for electronically comparing the 
score known to the quality-control personnel and 
the resolver score and for reporting a discrepancy 
between the known score and the resolver score . 



13 ^ A system for electronically providing performance 

feedback based upon a comparison of a test resolver 's 



Application/Control Number: 10/690,122 
Art Unit: 3600 



Page 8 



performance to predetermined performance criteria, the 
system comprising: 

a) receive means for electronically receiving 
information related to a first performance level 
from a first resolver and a predetermined 
performance level ; 

b) comparison means for electronically comparing the 
first performance level and the predetermined 
performance level; and . 

c) feedback means for electronically reporting a 
discrepancy based on predefined criteria between 
the first performance level and the predetermined 
performance level* 

14. The system of claim 13 wherein the feedback means 
comprises means for electronically reporting the 
discrepancy to the first test resolver. 



15. The system of claim 13 wherein the feedback means 
comprises means for electronically reporting the 
discrepancy to a system manager. 
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16. A system for electronically providing performance 

feedback based upon a comparison of a test resolver's 
performance to predetermined performance criteria, the 
system comprising: 

a) receive means for electronically receiving 
information related to a first test-score validity 
level from a first resolver and a predetermined 
validity level; 

b) comparison means for electronically comparing the 
first validity level and the predetermined 
validity level; and 

c) feedback means for electronically reporting a 
discrepancy based on predefined criteria between 
the first validity level and the predetermined 
validity level. 

17* The system of claim 16 wherein the receive means 
comprises means for receiving a test score for a 
quality item from the first resolver and a known score 
for the quality item. 

18. The system of claim 16 wherein the feedback means 
comprises means for electronically reporting the 
discrepancy to the first test resolver. 
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19. The system of claim 16 wherein the feedback means 
comprises means for electronically reporting the 
discrepancy to a system manager. 
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20. ^ A system for electronically providing performance 

feedback based upon a comparison of a test resolver 's 
performance to predetermined performance criteria , the 
system comprising: 

a) receive means for electronically receiving 
information related to a first reliability level 
from a first resolver and a predetermined 
reliability level; 

b) comparison means for electronically comparing the 
first reliability level and the predetermined 
reliability level; and 

c) feedback means for electronically reporting a 
discrepancy based on predefined criteria between 
the first reliability level and the predetermined 
reliability level. 

21. The system of claim 20 wherein the receive means 
comprises means for receiving during a first time 
period a first score from the first resolver for a test 
item and for receiving during a second time period a 
second score from the first resolver for the test item. 
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2ifc. The system of claim 20 wherein the feedback means 
comprises means for electronically reporting the 
discrepancy to the first test resolver. 



23. The system of claim 20 wherein the feedback means 
comprises means for electronically reporting the 
discrepancy to a system manager. 

24. A system for electronically providing performance 
feedback based upon a comparison of a test resolver 's 
performance to predetermined performance criteria, the 
system comprising: 

a) receive means for electronically receiving 
information related to a first speed level from a 
first resolver and a predetermined speed level; 

b) comparison means for electronically comparing the 
first speed level and the predetermined speed 
level; and 

c) feedback means for electronically reporting a 
discrepancy based on predefined criteria between 
the first speed level and the predetermined speed 
level. 
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25. The system of claim 24 wherein the receive means • 
comprises means for receiving at least one of the 
following: a first scoring rate for the first resolver, 
an average scoring rate from a selected group of 
resolvers, and a second scoring rate from a second 
resolver. 
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26. The system of claim 24 wherein the feedback means 
comprises means for electronically reporting the 
discrepancy to the first test resolver. 

27. The system of claim 24 wherein the feedback means 
comprises means for electronically reporting the 
discrepancy to a system manager. 

28. A system for increasing test resolver efficiency by 
monitoring test resolver performance and recommending 
break times to the test resolvers, the system 
comprising: 

a) receive means for electronically receiving at 
predetermined past intervals in time first 
performance levels from a test resolver and for 
electronically receiving a second predetermined 
performance level; 

b) comparison means for electronically comparing the 
second predetermined performance level and the 
first performance level; and 

c) recommendation means for electronically detecting 
a discrepancy based on predefined criteria between 
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the second predetermined perf orraance level and the 
f^rst performance level as indicated by the 
comparison means and for signalling the test 
resolver to halt test scoring for a break time. 

2 K 9. The system of claim 28 wherein the receive means 
comprises means for electronically receiving first 
validity levels from the test resolver and for 
electronically receiving a second predetermined 
validity level. 

30. The system of claim 28 wherein the receive means 
comprises means for electronically receiving first 
reliability levels from the test resolyjsr and for 
electronically receiving a second predetermined 
reliability level. 

31. The system of claim 28 wherein the receive means 
comprises means for electronically receiving first 
speed levels from the test resolver and for 
electronically receiving a second predetermined speed 
level . 
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32. The system of claim 28 wherein the recommendation means 
further comprises diversion means for electronically 
detecting a discrepancy based on predefined criteria 
between the second scoring rate and the first scoring 
rates as indicated by the comparison means and for 
providing the test resolver with diversionary 
activities . 

33. The system of claim 28 wherein the recommendation means 
further comprises resolver-selected diversion means for 
enabling the test resolver to select diversionary 
activities when the recommendation means has signalled 
the test resolver for the break. 

34. A system for increasing test resolver efficiency by 
monitoring test resolver performance, and recommending 
break times to the test resolvers, the system 
comprising: 

a) receive means for electronically receiving at 
predetermined past intervals in time first test 
scoring rates from a test resolver and for 
electronically receiving a second present scoring 
rate from the test resolver; 

b) comparison means for electronically comparing the 
second scoring rate and the first scoring rates? 
and 
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cj^ recommendation means for electronically detecting 
a discrepancy based on predefined criteria between 
the second scoring rate and the first scoring 
rates as indicated by the comparison means and for 
signalling the test resolver to halt test scoring 
for a break time. 



35. The system of claim 34 wherein the recommendation means 
further cd&prises diversion means for electronically 
detecting a discrepancy based on predefined criteria 
between the second scoring rate and the first scoring 
rates as indicated by the comparison means and for 
providing the test resolver with diversionary 
activities. 

36. The system of claim 34 wherein the recommendation means 
further comprises resolver-selected diversion means for 
enabling the test resolver to select diversionary 
activities when the recommendation means has signalled 
the test resolver for the break. 
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37. A system for preventing bias in the scoring of answers 
to tests taken by persons in different particular 
classifications , the system comprising: 
a) receive means for electronically receiving a 

plurality of answers to test questions, the test 
answers each corresponding to a test taken by a 
person from a particular definable classification 
and comprising an electronic representation of at 
least a portion of one or more test answer sheets; 
fc>) classification means for receiving a particular 
classification of each of the respondents or a 
group of the respondents and for electronically 
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associating each of the particular classifications 
with the test answers for the corresponding 
respondent or group; and 
c) routing means for electronically selecting a test 
resolver for each of the answers based upon the 
particular classification of the corresponding 
respondent or group and for transmitting the 
answers to the selected test resolver. 

38. The system of claim 37 wherein: 

a) the classification means comprises means for 
receiving a racial classification identifying a 
race of each of the test resolvers; and 

b) the routing means comprises means for 
electronically transmitting each of the test 
answers to a test resolver having a same, 
different, or preferred blend racial 
classification as the racial classification of the 
corresponding respondent or group. 



39. The system of claim 37 wherein: 

a) the classification means comprises means for 
receiving a geographical classification 
identifying a location of each of the test 
resolvers; and 
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b) 



the routing means comprises means for 
electronically transmitting each of the answers to 
a test resolver having a same, different , or 
preferred blend geographical classification as the 
geographical classification of the corresponding 



respondent or group. 



40. A system for electronically and dynamically providing 
scoring rules to test resolvers to assist the test 
resolvers in scoring test items to which the scoring 
rules apply, the system comprising: 

a) receive means for electronically receiving a 
plurality of answers for test questions, the 
answers each comprising an electronic 
representation of at least a portion of a test 
answer sheet; 

b) storage means for electronically storing a 
plurality of scoring rules, each of the scoring 
rules relating to a procedure for scoring at least 
a particular one of the test questions; 

c) display means for selectively presenting the test 
answers to a test resolver; and 



d) rules means for selectively presenting to the test 
resolver the scoring rules related to the test 
question corresponding to the answer presented to 
the test resolver. 
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And add the following claims: 

4 1 . (New) A method for electronically and dynamically providing evaluation rules to 
resolvers to assist the resolvers in evaluating data items to which the evaluation rules apply, the 
method comprising the steps of: 

a) receiving a plurality of data items over a distributed computer network; 

b) storing a plurality of evaluation rules, each of the evaluation rules relating to a 
procedure for evaluation at least a particular one of the data items; 

c) selectively presenting the data item to a resolver over a distributed computer network; 

and 

d) automatically selectively presenting to the resolver the evaluation rules related to the 
data item presented to the resolver over a distributed computer network. 

42. (New) The method of claim 41, wherein the step of presenting the data item and 
the step of presenting the corresponding evaluation rules are performed simultaneously. 

43. (New) A system for electronically and dynamically providing evaluation rules to 
resolvers to assist the resolvers in evaluating data items to which the evaluation rules apply, the 
system comprising: 

a) a receiving subsystem configured to a plurality of data items over a distributed 
computer network; 

b) a storing subsystem configured to store a plurality of evaluation rules, each of the 
evaluation rules relating to a procedure for evaluating at least a particular one of the data items; 

c) a presentation subsystem configured to selectively the data item to a resolver over a 
distributed computer network; and 
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d) a selection subsystem configured to automatically selectively to the resolvcr the 
evaluation rules related to the data item presented to the resolver over a distributed computer 
network. 

44. (New) The system of claim 43, wherein the presenting the data items and the 
presenting the corresponding evaluation rules occur simultaneously. 

45. (New) A computer readable program storage medium tangibly embodying a data 
package and associated verification instructions executable by a computer system to 
electronically and dynamically providing evaluation rules to resolvers to assist the resolvers in 
evaluating data items to which the evaluation rules apply by performing steps including: 

a) recei ving a plurality of data items over a distributed computer network; 

b) storing a plurality of evaluation rules, each of the evaluation rules relating to a 
procedure for evaluating at least a particular one of the data items; 

c) selectively presenting the data item to a resolver over a distributed computer network; 

and 

d) automatically selectively presenting to the resolver the evaluation rules related to the 
data item presented to the resolver over a distributed computer network. 

46. (New) The computer readable program storage medium according to claim 45, 
wherein the presenting the data items and the presenting the corresponding evaluation rules occur 
simultaneously. 



